1. Field of the Invention
The present invention applies selected features of advanced machine vision technology to the noninvasive and remote evaluation and quantification of livestock volumes and dimensions. These data are shown to be of value during breeding selections, feedlot evaluations, meat packing transactions and carcass evaluations.
2. Discussion of the Prior Art
Throughout history there has been the desire to measure domestic livestock. Whether such measurements included height, weight, width, length or strength, the measurement need was present. In the recent past weight alone was often used as an indicator of size and value. As consumer demand for leaner meats with lower fat content increases, the need grows to refine animal measurement techniques as well. The current industry trend is toward the consolidation of small operations into larger, more efficient operations. This trend requires not only accurate measurements, but automation and rapid data acquisition as well. The background of the present invention includes developments in both the fields of animal measurements and volumetric imaging.
1.1. Livestock Measurements
According to USDA statistics, U.S. commercial cattle slaughter totaled 35.4 million head in 2001 with commercial hog slaughter at 98.0 million head for the same year. At each stage of meat production there is a need to measure the volume and dimensions of the animals.
Breeding Evaluations.
In general, breeders of both cattle and hog populations are attempting to efficiently produce animals with a higher percentage of lean meat and a lower percentage of fat. However, in order to evaluate breeding efficacy, it is essential for feedlots and packing facilities to accurately measure and track live physical characteristics, growth and performance data and match these with end product carcass information. Without accurate and automated measurements and data collection for the individual animal, such tracking is not possible.
Feedlot Evaluations.
Ideally, the physical and growth characteristics of each animal should be known at each stage of its stay in the feedlot to achieve optimum management. However, in order for this optimum management to be implemented, the volume and physical dimensions of each animal must be accurately measured regularly during the animal's stay at the feedlot. Since North American feedlots may house ten thousand to one hundred thousand animals, it is essential that the aforementioned, repeated measurements be acquired accurately and rapidly as part of an automated tracking system.
Live-Animal, Slaughter Plant Evaluations.
According to USDA statistics for 2001, the top 15 cattle slaughter plants account for 57 percent of the total production. Similarly, those statistics for the top 12 hogs slaughter facilities account for 53 percent of the total. For each of the leading plants an average of these numbers yields an average weekly slaughter of 25,961 and 83,237 head for cattle and hogs, respectively. With production at these levels the measurement of live animals upon delivery requires not only accuracy, but also automation and rapid data acquisition.
Carcass Evaluations.
The same efficiency needs that pressure slaughter plants continue into the meat packaging stages. The accurate measurement and evaluation of carcasses is critical as feedback to breeders and feedlot operators to evaluate changes.
1.2. Livestock Technology
For the above reasons, non-contact imaging techniques are advantageous to measure animals and carcasses in order to achieve both measurement accuracy and improved acquisition speed. Much of the existing state-of-the-art for measuring animals or carcasses relies upon the acquisition of images showing silhouettes or profiles of an animal (or carcass). In any one view, this technique provides only a record of the target animal's shadow with a loss of any three-dimensional shape within the silhouette outline. In order to attempt volumetric measurements many viewing angles must be used. Even with multiple views from many angles, the resulting volume estimation is inaccurate when any type of surface concavity is present.
Other techniques include the use of visible-spectrum, video images to evaluate lean and fat content of the carcass. Since this video image is only two-dimensional (2D), any proportional analysis of fat verses lean meat can only be a calculation of the area in a 2D view of the carcass. Such analyses of three-dimensional volumes in only two dimensions are fraught with error.
Numerous inventors have contributed to the current state-of-the-art for the measurement of animals. Early patents in this field involved automated gates and weighing systems. One such patent was U.S. Pat. No. 4,280,488 (Ostermann) which describes a gate and chute system for sorting and segregating animals by weight using a scales as the measurement method. U.S. Pat. No. 4,288,856 (Linseth) shows a method for separating and grouping animals from a feedlot herd according to weight gain characteristics also using scale measurements. U.S. Pat. No. 4,617,876 (Hayes) describes an animal identification and control system which employs an identification tag which can be read from a distance, an automated weighing system and automated gates to control the movement of the animal.
Hayes, U.S. Pat. No. 4,745,472, proposes ways to obtain side and top profiles of animals via wall grids and visible-spectrum, video images. Chute mechanisms are used to position the animal in an upright, standing position. This patent also proposes ways of making area and linear measurements from these profiles which relate to physical characteristics of the animal.
Chevelier, et al., U.S. Pat. No. 5,194,036, present a method and apparatus for grading animal carcasses involving the use of video images of the carcasses. This patent requires somewhat complicated methods to rotate each carcass such that images can be obtained at multiple angles with multiple cameras. The two-dimensional, recorded images are then compared to a library of images in an attempt to achieve automated grading of the meat.
Petersen, et al., U.S. Pat. No. 4,939,574, presents a method and apparatus for obtaining a dark silhouette or contour of an animal carcass, particularly that of cattle. Details are provided for a light-screening chamber into which the carcass is placed, consisting of one wall with a lighted surface opposite a wall of frosted glass onto which the carcass shadow falls. The shadow or dark silhouette is recorded and digitized via a visible-spectrum, video camera placed at a distance behind the frosted glass wall. The video image is digitized and analyzed via a computer system. Front lit images of the carcass are also recorded and superimposed over the silhouette outline in an attempt to identify the lean (red) and fat (white) portions of the carcass image. Additional provisions are made for the use of manual probes to measure fat thickness at various locations in an attempt to improve the estimate of the fat and lean content.
O'Brien, et al., U.S. Pat. No. 5,205,799, describes a three-dimensional, stereoscopic, imaging system used in conjunction with an X-ray system to view the exterior and interior of an animal carcass.
Bamjii, U.S. Pat. No. 6,323,942, describes a 3D image sensor employing a two-dimensional array of pixel light sensing detectors and dedicated electronics fabricated on a single CMOS integrated circuit. This invention emits a pulse of light and times the response time for each pixel in an N.times.M optical detection array. Measuring the time-of-flight (TOF) for the emitted light to travel to a target and return to given pixel permits a distance to be computed for each part of the reflected image. The proposed TOF measurements are presented as either individual high-speed counters for each pixel or individual gated charge accumulator for each pixel.
Faulkner, U.S. Pat. No. 5,335,288, uses biometric measurements of hand silhouette and finger height to identify a person. The finger height is measured via a structured light technique.
In U.S. Pat. No. 5,412,420, inventor James S. Ellis presents a three-dimensional (3D) measurement system for animals. The patent discloses the use of LASAR cameras in a system which appears to employ an amplitude-modulated, phase-detection, time-of-flight laser technique similar to the Lidar scanning system described in U.S. Pat. No. 5,006,721.
Ellis U.S. Pat. No. 6,777,353 describes a measurement system which purportedly measures the three-dimensional linear, angular and volumetric characteristics of an animal or carcass, such as beef cattle. The capacity to obtain these useful livestock measurements relies largely upon the system's ability to accurately determine the three-dimensional surface of the target animal. This patent teaches that the three-dimensional surface is determined by first projecting light spots onto the surface of the target animal as shown in FIG. 1 of the patent. These light spots are then observed by the receiving camera located immediately adjacent to the projector as shown in FIG. 6 of the patent. According to this patent, the image obtained by the receiving camera may be analyzed to determine the dimensions of the light spots on the surface of the target animal. As described in column 3, lines 56–65, the measured diameter of a given light spot, as measured in the camera image, is proportional to the distance between the target surface and the receiving camera. A spot diameter of 1.5 inches corresponds to a distance of 6 feet, while a spot diameter of 1.75 inches corresponds to a distance of 7 feet. Variations of structured light which may include squares, vertical bars or horizontal bars behave in a manner similar to the light spots.
Jones, et al., U.S. Pat. No. 5,458,418, describes a method for detecting poor meat quality from thermal images of cattle and swine. If the thermal images reveal skin temperatures outside of the predetermined, absolute temperature ranges, 28–32+/−2 degrees C. for cattle and 24–26+/−2 degrees C. for swine, the animal is likely to provide poor meat quality.
Hurnick, et al., U.S. Pat. No. 5,474,085, have proposed a thermographic imaging system for remote sensing of various characteristics of livestock, such as weight, carcass pH, temperature and location.
Scofield, U.S. Pat. No. 5,483,441, has proposed a method for evaluating changeable configuration bodies which move through first and second scenes, corresponding to differing first and second fields of view. The Scofield patent describes methods of obtaining top views, side views, profiles and outline measurements using common, visible-spectrum, video cameras similar to a number of other inventors. This conclusion is especially evident in the embodiment section, column 12, line 59 through column 13, line 10, where a silhouette is created with the animal black and the background white.
Scofield et al., U.S. Pat. No. 5,576,949, is similar to U.S. Pat. No. 5,483,441 with the addition of black and white bars on the walls of the chute to provide a repetitive pattern which is detected via spectral analysis in order to help identify that part of the image that is background and not animal. Apparently this approach provides a more reliable silhouette.
Tong, et al., U.S. Pat. No. 5,595,444, improves upon the invention of U.S. Pat. No. 5,58,418 (Jones, Schaefer, Tong, Scott, Gariepy, and Graham) for identifying cattle and swine that are likely to provide poor meat quality. This invention acquires thermal images of the animals and identifies those that statistically fall outside of a range of normal for a given herd or group as those likely to provide poor meat quality. This is likely to be more accurate and also allows for extreme ambient temperatures since the group statistics would shift with the ambient temperatures.
Scofield, et al., U.S. Pat. No. 5,644,643, is a continuation of U.S. Pat. Nos. 5,483,441 and 5,576,949. This continuation contains additional claims regarding the chute construction and appearance to provide better contrast with regards to the animal.
In U.S. Pat. Nos. 5,673,647, 6,000,361, 6,135,055, and 6,318,289 Bi, William C. Pratt describe cattle management systems in great detail. These system include individual animal identification by electronic methods, animal measurement, automated data inputs, cattle handling and sorting components, computer systems to calculate the optimui slaughter weight and marketing date for shipment of the animal to a meat processing plant. These patents also include a description of computer calculations for correlating live animal characteristics to the measured carcass characteristics.
Godik, U.S. Pat. No. 5,699,797, deals with the properties of human skin obtainable via IR imaging. It employs an active IR illumination source and IR detectors. While applicable to thermal imaging of skin, it discusses skin penetration of 1 cm or less.
Tong, et al., U.S. Pat. No. 5,944,598, uses infrared thermography to detect poor meat quality in live animals. With their technique thermal images are acquired of a group of animals. A mean temperature is computed for the group and every animal in the group. Animals are rejected as having a high probability of producing poor meat quality if their individual thermal characteristics differ significantly from that of the group.
Anderson, et al., U.S. Pat. No. 6,032,084, proposes a fully-automated, animal feedlot management system where each feed delivery vehicle uses real-time virtual reality modeling and satellite-based, global positioning system (GPS) technology to direct various type of feedlot operations. Though the proposed automation is abundant, there is no feedback measuring the growth and performance of the animals. The present invention fills that void by accurately.
Schaefer et al., U.S. Pat. No. 6,123,451, presents a process for determining a tissue composition characteristic of an animal via infrared thermographic imaging. This invention involves the acquisition of a thermal image of the animal and/or carcass, calculating statistical information from the thermal image, inputting the statistical information into a predictive model, and solving the predictive model for tissue composition characteristics such as lean body mass, carcass fat composition, and lean yield. Correlation data presented in the patent provided correlation coefficients between thermal properties and stated variables that ranged from r=00.94 (r2=0.89) to r=0.72 (r2=0.52). The IR images for these data were obtained via a 2D thermal imaging camera.
Belk, et al., U.S. Pat. No. 6,198,834 B1, proposes an image analysis system for scoring characteristics that predict the palatability and yield of a meat carcass or cut. Specifically, the illustrative embodiments included color and color variability of fat and lean tissue, extent of marbling, average number and variance of marbling flecks per unit area, average size of marbling, the variance of marbling size, average texture of marbling and lean tissue, firmness of lean tissue, density of lean tissue, and density of connective tissue.
Cureton, U.S. Pat. No. 6,216,053 Bi, discloses a fully automated feedlot management system similar to that described in U.S. Pat. No. 6,032,084.
1.3. Volumetric Measurements
Numerous methods are available for the general computation of volume and the dimensional measurement of an object. In general, the process of generating 3D, volumetric data contains the following steps:
1. Determine the necessary number and direction of views based upon the complexity of the shape;
2. Acquire a three-dimensional surface image from each view;
3. Generate a 3D surface via mesh or other techniques;
4. Register the multiple surfaces.
5. Fuse the surfaces into one geometry;
6. Display the data; and
7. Compute measurements based upon the 3D model.
The complexity of the target volume determines the number and direction of views required. Simple convex volumes may require as few as two views to image the entire shape. Complex volumes with deep indentations may require multiple views of a single region. The non-contact acquisition of 3D surfaces may be accomplished with a number of technologies. Regardless of method, each surface provides a three-dimensional representation of the volume's shell as viewed from the direction of the given camera. After the acquisition of multiple surfaces, it is necessary to register the surfaces in a common coordinate system. This is most easily accomplished if the geometric relationship between cameras is fixed and known. After registration, the multiple surfaces may be fused into a common volume or geometry. This step must take into account the direction of each view and any loss of accuracy near the edges of the 3D surfaces. Once geometric fusion has been accomplished, a 3D triangulated mesh may be generated for the volume surface. This mesh mathematically represents the surface of the volume and enables the numerical calculation of volume that is desired. Once the volume has been calculated, it may be displayed graphically or numerically. It may also be used as input or feedback for a manufacturing process.
1.4. Three-Dimensional Surface Measurements Technologies
Common to many of these methods is the computation of 3D surfaces. FIGS. 1-1 and 1-2 depicts the many technologies which may be employed to obtain three-dimensional surface measurements. Each has advantages and disadvantages for a given application. The conditions associated with the measurement of live or carcass cattle and hogs makes many of these general techniques impractical.
The large number of animals necessitates an automated measurement system which acquires, processes and records the measurement data rapidly. In a slaughter plant situation, an animal may be slaughtered every 3 to 10 seconds. A lengthy measurement process is not acceptable. Additionally, live animals are often moving. Even carcasses are in constant motion on an overhead conveyor belt. To achieve an accurate measurement an apparatus must be capable of freezing such movement. The technologies represented in FIGS. 1-1 and 1-2 need to be examined in light of light of the specific requirements for measuring live and carcass cattle and hogs.
Contact vs. Non-Contact.
Contact technologies are not well-suited for livestock measurements. Contact methods typically employ a precision, mechanical arm with a sensitive tip. This assembly is carefully scanned over the object acquiring data points one at a time. They can take up to several hours for scanning and digitizing one object. While they can be very accurate, they are best-suited for digitizing small solid objects. Additionally, the precision arm and sensitive tip are not well-suited for a livestock environment.
Conversely, non-contact methods are much more likely to be a match for this application since data acquisition may occur rapidly from a distance. The sensitive equipment can be located in a safe location somewhat removed from the livestock environment.
Reflective Optical vs. Non-Optical.
Reflective methods which employ optical technology can be used successfully for acquiring 3D livestock data. Reflected light methods include those that employ structured illumination patterns to achieve specific signal processing advantages. Non-optical, reflective methods, such as sonar or imaging radar are not as good a match for this application. Sonar or other ultrasonic methods typically use a liquid coupling medium, which is not practical for this project. While ultrasonic air operation is possible, the efficiency and lateral resolution present significant technical challenges. Imaging radar is typically used to map the earth's surface. While well-suited for large targets such as a mountain range, imaging radar is not likely to provide sufficient resolution for this project.
Emitted vs. Transmissive.
Non-contact, emitted technologies include primarily infrared (IR) methods. While IR in the 8–12: μ wavelength is useful for imaging thermal data, the use of structured light techniques in this band is difficult. Thermal patterns are much more difficult to generate and project than optical patterns. Additionally, the resolution of thermal patterns is substantially less than their optical counterparts. Presently, IR imaging systems are slower, more expensive, and of lower resolution than optical systems. Transmissive optical methods are not applicable since cattle and hogs are not transparent to light. Transmissive X-ray systems are not being considered since they render muscle and soft tissue nearly invisible and cost and safety are significant factors.
Optical Methods—Active vs. Passive.
Non-contact, reflective, optical methods for obtaining 3D data may be further divided into active and passive systems. Passive systems rely on the ambient light and surface texture of the target to provide sufficient information to compute dimensional data. Passive methods include passive stereo, shape from shading, shape from silhouette, passive depth from focus, and passive depth from defocus. Since passive system depend on ambient conditions, their reliability is often uncertain. Active optical methods typically employ a controlled light source of some kind which greatly increases system reliability over the similar passive system without the active source.
Active Optical Methods.
Active optical systems include pulsed or modulated light, interferometry, active depth-from-focus, active depth-from-defocus, active silhouette, active triangulation, and active stereoscopic.
Pulsed light methods utilize a pulse of light which bounces off of the target surface and returns to the source. The round trip time is measured and the distance computed from knowledge of the speed of light. A variation on this principle employs a continuously modulated light beam which is bounced off of the target and returned to the source. The phase of the received signal is demodulated to determine the time delay associated with the round trip to the target and back. Both of these variations require expensive test equipment to measure the small time delays inherent in light propagation. A spot or stripe scanning process is also required.
Interferometry methods include moire patterns and holography. Such methods are not applicable to this application since they work best when the surface depths are microscopic.
Active depth-from-defocus (DFD) technology take advantage of the commonly observed fact that objects in focus appear crisp and detailed, while objects out of focus appear blurred. Under controlled, structured light conditions it is possible to measure the degree of blurring and thus compute the associated distance between a given image spot and the reference distance where the image is in complete focus.
Active depth-from-focus (DFF) utilizes similar principles to DFD. However, DFF requires a focal scan through the range of the target. A multitude of images are acquired and processed to identify the precise distance at which each surface point is in the best focus. Since magnification changes with focal distance in most optical systems, registration and alignment of the multiple images can be a problem. During the multiple image acquisitions, animal movement can also be a problem for this application.
Active triangulation typically uses laser spot scanning or scanning stripes. In this method the laser beam and the visible-spectrum camera are at different Such scanning system require multiple images and frequently long scanning times. Computer memory requirements and image processing times can be significant. Consider a CCD camera acquiring images at a video rate of 30 images per second. With a 640×480 pixel image (307,200 pixels per image) and only an 8-bit pixel depth, data is acquired at a 73.7 Mb per second rate. Additionally, holes in the computed surface result when a surface feature obstructs either the laser beam or the camera view.
Active stereoscopic vision systems may also be used to obtain 3D surface measurements. This method uses two cameras separated by a distance sufficient to triangulate on a given point on the target surface. A minimum of two images are required to compute the target surface. Holes in the computed surface result when a surface feature obstructs one of the camera views.
Active silhouette (or profile) is sometimes considered an active optical system. Since it only acquires the outline or shadow of the target, it is not a 3D measurement.
Patent Review for Active Depth-from-Defocus and Depth-from-Focus Technologies
Since the volumetric measurement of live and carcass cattle and hogs imposes numerous constraints on 3D surface measurement technologies, a review of applicable patents will focus on those technologies which most closely meet the of applicable patents will focus on those technologies which most closely meet the requirements of this application, namely active depth-from-defocus (DFD) and active depth-from-focus (DFF) methods.
TABLE 3Patent Review - Volumetric Measurements via Focus/Defocus TechniquesU.S.Pat.No.TitleInventorAssigneeComments6,269,197Determining aAaron S. WallackCognexdepth usingdepthCorp.defocus & contrastmeasurements from3 images withstructuredillumination6,219,461Determining aAaron S. WallackCognexdepth usingdepthCorp.defocus & differentstructuredilluminationpatterns for each ofmultiple 2D images6,148,120Warping ofMichaelCognexcorrectsfocal images toSussmanCorp.correspondencecorrecterrors amongcorrespondencemultiple imageserrorwhen focaldistances (&magnification) arechanged with non-telecentric opticalsystems - useful infocus & defocussystems6,025,905System forMichaelCognexmethod forobtaining aSussmanCorp.obtaining a uniformuniformillumination imageilluminationfrom multiplereflectancestructuredimage duringillumination imagesperiodic(good backgroundstructureddiscussion)illumination5,912,768Depth-from-BradleyCognexa depth-from-defocus opticalSissom,Corp.defocus opticalapparatus withMichaelapparatus for 3Dinvariance toSussmanimaging; includessurfaceilluminationreflectancesource, projectionpropertieslens, viewing lens,and beam splitter(good backgrounddiscussion)5,878,152Depth fromMichaelCognexdepth of focusfocal gradientSussmanCorp.techniquesanalysis usingemploying albedoobject texturenormalization -removal byremoval of thealbedoreflectance effectsnormalizationof the object'snatural surfacetexture such thatonly the structuredlight illumination isobserved (goodbackgrounddiscussion)5,953,126OpticalJames M. ZavislanLucid Inc.a spot scanningprofilometrysystem which usesa measurement ofspot defocus toobtain a depthcalculation insteadof refocusing thelens assembly ateach new spotposition to obtainthe depth5,360,970Apparatus andDavid B. KayEastmanuse of a diffractionmethod for aKodak Co.grating to aidsingle returnfocusing of a laserpath signalon a data tracksensor system5,900,975Ghost imageMichaelCognexa plate beamsplitterextinction in anSussmanCorp.with polarizingactive rangefilter(s) whichsensoreliminates ghostimages used infocus/defocusimaging systemswith telecentricoptics5,300,786Optical focusTimothy A. Brunner,IBMan optical systemphase shift testMichael S. Hibbs,projecting phase-pattern,Barbara B. Peck,shifted, projectionmonitoringChristopher A. Spencepatterns onto asystem andsurface to quantifyprocessthe degree of focus -used withphotolithographictechniquesassociated withsemiconductormasks5,231,443AutomaticMuralidharaResearchA method based onranging andSubbaraoFoundationimage defocusautomaticofinformation forfocusingStatedetermining theUniversitydistance of objectsof NewYork4,841,325AutomaticKunihishNikonAn automaticfocusingHoshinoCorporationfocusing device fordevice forYoshinaridetecting thecameraHamanishi Kenamount defocusUtagawa4,088,408Device forErnest E. Nurcher,USA,A device formeasuring theStephen J. Katzberg,NASAmeasuring thecontour of aWilliam I. Kelly,contour of a sufacesurfaceIV
Discussion of Critical Patents:
U.S. Pat. No. 6,269,197—Determining a Depth
Abstract:
A three dimensional image is derived from two-dimensional images. At least one of the two-dimensional images has a predetermined number of pixels. Depth measurements are derived from the two-dimensional images. The number of derived depth measurements is substantially equal to the predetermined number of pixels. The three-dimensional image is derived from the two-dimensional digital images and the depth measurements.
Inventors: Wallack; Aaron S. (Natick, Mass.) Assignee: Cognex Corporation (Natick, Mass.) Issue date: Jul. 31, 2001 Discussion:
This patent is a division of the patent application associated with U.S. Pat. No. 6,219,461.
This invention relates to determining a depth or range sensing via defocus methods. It is intended for industrial applications such as solder paste volumes, 3D clay models, and inspection of semiconductor packages. It attempts to address the perceived state-of-the-art which allows 3D images to be derived from 2D images by exploiting optical principles related to the distance between an out-of-focus point and an in-focus point.
The essence of this invention is a method of analyzing pixel information in 2D images of a 3D object to obtain 3D surface information about the object. It uses at least three different images of the object at different optical path lengths, each with a different structured illumination projected onto the object surface. From this procedure contrast measurements in the 2D image are converted into depth calculations for the image. The structured illumination may be moved for each of the images.
U.S. Pat. No. 6,219,461—Determining a Depth
Abstract:
A three-dimensional image is derived from two-dimensional images. At least one of the two-dimensional images has a predetermined number of pixels. Depth measurements are derived from the two-dimensional images. The number of derived depth measurements is substantially equal to the predetermined number of pixels. The three-dimensional image is derived from the two-dimensional digital images and the depth measurements.
Inventors: Wallack; Aaron S. (Natick, Mass.) Assignee: Cognex Corporation (Natick, Mass.) Issue date: Apr. 17, 2001 Discussion:
This invention relates to determining a depth or range sensing via defocus methods and is a division of the patent application associated with U.S. Pat. No. 6,269,197.
The essence of this invention is a method of analyzing pixel information in 2D images of a 3D object to obtain 3D surface information about the object. For each of the 2D images a different structured illumination pattern is employed. The positions of a periodic structured pattern, with respect to the subject, are shifted by a portion of the repetition period. Focus-based depth measurements are derived for each pixel of at least one of the 2D images.
U.S. Pat. No. 6,148,120—Warping of Focal Images to Correct Correspondence Error
Abstract:
The invention corrects correspondence error among multiple images taken at different focal distances with non-telecentric optical systems, and is particularly useful in focal gradient analysis range imaging systems.
Inventors: Sussman; Michael (Winchester, Mass.) Assignee: Cognex Corporation (Natick, Mass.) Issue date: Nov. 14, 2000 Discussion:
This invention relates to 3D machine vision which employs depth-from-focus and depth-from-defocus techniques.
U.S. Pat. No. 6,025,905—System for Obtaining a Uniform Illumination Reflectance Image During Periodic Structured Illumination
Abstract:
The invention provides an apparatus and method for obtaining a uniform illumination reflectance image of an object, even though the object is illuminated only using periodic structured illumination. The uniform illumination reflectance image so-produced has precise geometric and photometric correspondence with images produced using the periodic structured illumination. To obtain the uniform illumination reflectance image, a sum of a spanning set of periodic structured illumination images is computed. The resulting summation image bears substantially no trace of periodic structured illumination. Various embodiments of the apparatus of the invention are disclosed employing illuminator motion, object motion, and ray deflection to obtain a plurality of periodic structured illumination images of different phase. The invention is useful with triangulation ranging systems using a striped periodic illumination mask, with depth-from-focus ranging systems, and with depth-from-defocus ranging systems.
Inventors: Sussman; Michael (Winchester, Mass.) Assignee: Cognex Corporation (Natick, Mass.) Issue date: Feb. 15, 2000 Discussion:
This invention relates to machine vision systems that employ periodic structured illumination. In applications which use structured illumination it is advantageous to use a uniform illumination image to normalize reflections from the target surface or distortions due to lens. This invention combines periodic structured illumination patterns in a manner that cancels out the periodic structures resulting in a uniform illumination. The cancellation typically consists of spatially shifting the illumination pattern by a specific phase of the illumination period.
U.S. Pat. No. 5,912,768—Depth-from-Defocus Optical Apparatus with Invariance to Surface Reflectance Properties
Abstract:
A depth-from-defocus optical apparatus is provided for use with a depth-from-defocus three-dimensional imaging system for obtaining a depth image of an object. The invention facilitates the formation of depth images of objects exhibiting specular reflection, either alone or in combination with diffuse reflection, thereby allowing the application of depth-from-defocus three-dimensional imaging to objects such as microelectronic packages. The optical apparatus of the invention generally includes an illumination source, a projection lens assembly for converging rays of incident light towards an object, and a viewing lens assembly for converging rays of reflected light towards an image plane. Importantly, the viewing lens assembly is of the same working f-number as the projection lens assembly. In preferred embodiments, both the projection lens assembly and the viewing lens assembly exhibit object-side telecentricity so as to substantially eliminate vignetting of off-axis specular object features, and consequently, substantially eliminate specular false depth. The invention can also include an uncrossed polarizer/analyzer pair to balance the dynamic range of specular reflections with the dynamic range of diffuse reflections so as to effectively utilize the limited dynamic range of a single image detector.
Inventors: Sissom; Bradley (Norwood, Mass.); Sussman; Michael (Winchester, Wash.) Assignee: Cognex Corporation (Natick, Mass. Issue date: Jun. 15, 1999 Discussion:
This invention relates to 3D machine vision which employs depth-from-focus and depth-from-defocus techniques. The components of this invention include an illumination source, a projection lens assembly, a viewing lens assembly, and a beamsplitter device. These components together make up a telecentric optical system for focal gradient range systems.
U.S. Pat. No. 5,878,152—Depth from Focal Gradient Analysis Using Object Texture Removal by Albedo Normalization
Abstract:
The invention provides a method and apparatus for obtaining a range image of an object. The method includes the act of “albedo normalization”, i.e., removing the effects of object reflectance using a structured illumination image of the object and a uniform illumination image of the object to provide an albedo-normalized image. This image is then processed using a focus measure to provide a focal image, which image is then used to provide a range image. The invention substantially removes the effects of object reflectance from an image acquired its degree of focus/defocus remains. Albedo normalization is achieved by dividing an image of an object taken under structured illumination by a corresponding image of the object taken under uniform illumination. The albedo normalization act removes the primary source of noise in range images obtained using a depth from defocus or depth from focus of structured illumination technique, by removing spurious image frequencies from the image before processing by a focus measure. The albedo normalization act permits the depth from defocus and depth from focus techniques to be used for one or more focal positions, and over a broad range of materials of interest in machine vision.
Inventors: Sussman; Michael (Winchester, Mass.) Assignee: Cognex Corporation (Natick, Mass.) Issue date: Mar. 2, 1999 Discussion:
This invention relates to machine vision systems which have the ability to provide range images of 3D objects via defocus methods using structured lighting.
This invention removes the effects of surface reflections from the object targeted with the structured illumination pattern. The natural object reflectance texture, also called ‘albedo’, may be eliminated by dividing the structured-illumination image by an image obtained under uniform illumination. This process is referred to as albedo normalization. The result is an image of the object which is dependent entirely on the structured illumination. This process is advantageous in applications such as depth from focus/defocus, laser triangulation, stereo vision, advantageous in applications such as depth from focus/defocus, laser triangulation, stereo vision, and other structured lighting methods.
U.S. Pat. No. 5,953,126—Optical Profilometry
Abstract:
A scanning reflection profilometry system utilizes an objective lens which focuses a beam at the surface under test and measures the profile of the surface (its height variations) in accordance with the amount of defocus of the reflected beam. Surface profile distortion which is focus dependent is reduced through the use of a transparent mask over the aperture of the lens in the path of the beam which is incident on and reflected from the surface under test and which covers a portion but not all of the aperture. A photodetector upon which the reflected beam is incident provides output signals representing the change in profile. The system has height sensitivity characteristic of a small spot size on the surface without signal distortion attributable to the diffraction anomalies associated with small spot sizes. A microprofilometer head having the objective lens and other optics is mounted on flexures and driven to execute reciprocal movement so as to scan the surface under test.
Inventors: Zavislan; James M. (Pittsford, N.Y.) Assignee: Lucid Inc (Henrietta, N.Y.) Issue date: Sep. 14, 1999 Discussion:
This invention uses defocus information to obtain a range image. However, it employs a single spot from a laser beam rather than a structured illumination pattern.
U.S. Pat. No. 5,360,970—Apparatus and Method for a Single Return Path Signal Sensor System
Abstract:
The radiation resulting from interaction with a data track or groove on a storage surface of an optical information storage and retrieval system is separated into three components and detected to provide tracking, focusing, and data signals. The separation is performed using a dual diffraction grating in a single optical path. The division between grating elements in the dual diffraction grating is oriented perpendicular to the data track or groove projected on the grating element. Diffraction radiation components generated by the dual diffraction grating are applied to a first and a second dual sensor elements. The first and second dual sensor elements provide a focusing signal. The undiffracted radiation component transmitted by the dual grating is applied to a third dual sensor. The division between sensors of the third dual senor is perpendicular to the division of the dual grating. Signals from the third dual sensor elements provide the tracking signal and the data signal. Several embodiments of the basic configuration are disclosed including a variety of configurations for defocusing the undiffracted transmitted radiation on the third dual sensor. In addition, a cylindrical lens can be used to defocus the radiation components from the diffraction grating in a single dimension.
Inventors: Kay; David B. (Rochester, N.Y.) Assignee: Eastman Kodak Company (Rochester, N.Y.) Issue date: Nov. 1, 1994 Discussion:
U.S. Pat. No. 5,900,975—Ghost Image Extinction in an Active Range Sensor
Abstract:
An apparatus is provided that includes a plate beamsplitter having a first surface coated with a partially reflective coating, and a second surface coated with an anti-reflective coating, and a polarizing filter, oriented with respect to the plate beamsplitter so as to substantially block light of substantially incompatible polarization that has traversed the plate beamsplitter, has been reflected by the object to be range imaged, and has been reflected by the plate beamsplitter towards the polarizing filter, thereby substantially preventing the formation of a ghost image of the object to be range imaged. Thus, the invention does not suffer from optical ghost images which commonly occur due to imperfect anti-reflection coatings used to make plate beam splitters. Also, the invention makes practical the use of plate beam splitters in depth from defocus and depth from focus range imaging systems employing coaxial active illumination and viewing.
Inventors: Sussman; Michael (Winchester, Mass.) Assignee: Cognex Corporation (Natick, Mass.) Issue date: May 4, 1999 Discussion:
U.S. Pat. No. 5,300,786—Optical Focus Phase Shift Test Pattern, Monitoring System and Process
Abstract:
A photolithography mask structure having a novel optical focus test pattern is described. The mask structure has a non-phase-shifted, transparent substrate and includes a phase shifter of other than 180E disposed between spaced, parallel opposing lines such that an alternating pattern of non-phase-shifted material and phase-shifted material is defined transverse said parallel lines. When projected onto the surface of an object measurable shifts of the test pattern corresponds in direction and magnitude with the extent of system defocus. Various alternating test pattern embodiments are presented, all of which include at least one phase shift window of other than 180E in relation to the mask substrate. Further, a monitoring system and a monitoring process are discussed employing the presented mask structures.
Inventors: Brunner; Timothy A. (Ridgefield, Conn.); Hibbs; Michael S. (Westford, Vt.); Peck; Barbara B. (Westford, Vt.); Spence; Chrisopher A. (Westford, Vt.) Assignee: International Business Machines Corporation (Armonk, N.Y.) Issue date: Apr. 5, 1994 Discussion:
U.S. Pat. No. 5,231,443—Automatic Ranging and Automatic Focusing
Abstract:
A method based on image defocus information is disclosed for determining distance (or ranging) of objects from a camera system and autofocusing of camera systems. The method uses signal processing techniques. The present invention includes a camera characterized by a set of four camera parameters: position of the image detector inside the camera, focal length of the optical system in the camera, the size of the aperture of the camera, and the characteristics of the light filter in the camera. In the method of the present invention, at least two images of the object are recorded with different values for the set of camera parameters. The two images are converted to one-dimensional signals by summing them along a particular direction whereby the effect of noise is reduced and the amount of computations are significantly reduced. Fourier coefficients of the one-dimensional signals and a log-by-rho-squared transform are used to obtain a calculated table. A stored table is calculated using the log-by-rho-squared transform and the Modulation Transfer Function of the camera system. Based on the calculated table and the stored table, the distance of the desired object is determined. In autofocusing, the calculated table and the stored table are used to calculate a set of focus camera parameters. The camera system is then set to the focus camera parameters to accomplish autofocusing.
Inventors: Subbarao; Muralidhara (Port Jefferson Station, N.Y.) Assignee: The Research Foundation of State University of New York (Albany, N.Y.) Issue date: Jul. 27, 1993 Discussion:
U.S. Pat. No. 4,841,325—Automatic Focusing Device for Camera
Abstract:
An automatic focusing device for use in camera lens systems comprises lens means such as a zoom lens for forming the image of an object, detecting means for detecting the amount of defocus of the image of the object formed by the lens means from a predetermined plane such as a film surface, memory means for storing at least one value of conversion coefficient and at least one value of correction coefficient which is used in a calculation for correcting the conversion coefficient, calculating means for correcting the conversion coefficient in accordance with the amount of defocus and the correction coefficient and for calculating the driving amount of at least a portion of the lens means on the basis of the corrected conversion coefficient and the amount of defocus, and lens driving means for driving at least a portion of the lens means, e.g., the front lens group of a zoom lens, in accordance with the driving amount calculated by the calculating means. Disclosed also a lens system, as well as a camera, incorporating this automatic focusing device.
Inventors: Hoshino, deceased; Kunihisa (late of Tokyo, JP); Hamanishi; Yoshinari (Tokyo, JP); Utagawa; Ken (Kawasaki, JP) Assignee: Nikon Corporation (Tokyo, JP) Issue date: Jun. 20, 1989 Discussion:
U.S. Pat. No. 4,088,408—Device for Measuring the Contour of a Surface
Abstract:
The invention is a device for measuring the contour of a surface. Light from a source is imaged by a lens onto the surface which concentrates the energy from the source into a spot. A scanning means is used to scan the spot across the surface. As the surface is being scanned the surface moves relative to the point of perfect focus. When the surface moves away from perfect focus the spot increases in size, while the total energy in the spot remains virtually constant. The lens then re-images the light reflected by the surface onto two detectors through two different sized apertures. The light energy going to the two detectors is separated by a beam splitter. This second path of the light energy through the lens further defocuses the spot, but as a result of the different sizes of the apertures in each light detector path, the amount of defocus for each is different. The ratio of the outputs of the two detectors which is indicative of the contour of the surface is obtained by a divider.
Inventors: Burcher; Ernest E. (Newport News, Va.); Katzberg; Stephen J. (Yorktown, Va.); Kelly, IV; William L. (Hampton, Va.) Assignee: The United States of America as represented by the Administrator of the (Washington, D.C.) Issue date: May 9, 1978.
1.5. Surface and Volumetric Renderings
There are many methods for visualization of volume data. A complete description of this large and rapidly changing field is beyond the scope of this discussion. However, two popular approaches are surface rendering and volume rendering. Surface rendering is a technique which treats the volume as having only a combination of surfaces or shells. Volume rendering on the other hand, maintains and manipulates many cubic building block known as ‘voxels’ to represent the volume. Volume rendering may be especially useful when the entire volume of the object contains information (density, elasticity, acoustic impedance) such as with magnetic resonance or ultrasound images. Both methods may begin with a 3D point cloud of data points as might be obtained from one or more range images.
Surface Rendering
In surface rendering the volumetric data must first be converted into geometric primitives, by techniques such as isosurface extraction or isocontour extraction. These primitives, such as polygon meshes or contours, are then rendered for display using conventional display techniques.
Advantages of Surface Rendering Include:
a) fast display and manipulation of the 3D reconstructions since only the surface vertices need to be manipulated and stored.
Disadvantages of Surface Rendering Include:
a) a required intermediate conversion to a surface representation which can sometimes be quite complex;
b) the lack of internal details of the volumes, since only the surfaces or shell is maintained; and
c) susceptibility to discontinuities in the 3D scanning.
One common method to determine a surface from a set of discrete data points is known as the Marching Cube Algorithm. This algorithm is a table-based, surface-fitting algorithm for rendering surfaces in volume space. The basic idea is to march a cube through the volume containing the surface to determine if the cube, in a given position, is totally inside the surface, totally outside the surface, or intersecting the surface. For those cube positions intersecting the surface, an index is maintained which records which of the 8 cube vertices (corners) are inside the surface and which vertices are outside the surface. Theoretically, 28=256 combinations are possible. However, eliminating symmetrical and inverse duplications, 14 unique configurations exist. Each configuration of vertices which are within the surface and vertices which are outside the surface results in a specific shape or surface patch bounded by the shape of the marching cube. Surface planes intersecting near a cube corner result in a triangular surface intersection, while surface planes which intersect four sides of the cube results in a surface patch having a rectangular shape. When the cube has completed its march through the volume, the resulting index of intersecting cube positions and the record of which vertices where inside and outside the surface can be used to create a patchwork quilt which is an accurate representation of the surface. Even greater surface resolution is possible if interpolation is used to determine where the surface intersects each cube edge as the cube progresses through the volume. The end result is a table of surface patches which can be passed to a rendering program that maps them into image space.
Another common method to obtain a surface from a set of discrete 3D surface points is known as Delaunay Triangulation. In this technique a set of lines is generated connecting each point in 3D space to its natural neighbors. The result is a triangular mesh, with non-uniform vertex locations, which represents the surface. If desired, this surface can then be converted to a rectangular mesh grid via resampling and interpolation. Such a rectangular mesh grid is easily displayed using common graphics programs.
Contour algorithms may also be used to convert non-uniformly sampled, discrete 3D surface data into a surface portrayed on a rectangular grid. In this type of algorithm lines are drawn through or between existing 3D data points of equal elevation. This series of lines may resemble the rings on a topographical map of a mountain. These equi-planar lines may in turn be converted to a rectangular mesh grid via resampling and interpolation.
Volume Rendering
In volume rendering the volumetric data is sampled into many cubic building blocks called ‘voxels’ (volume element), the volumetric equivalent to the 2D ‘pixel’ (picture element). Each voxel carries one or more values for characteristics of the volume such as color, density, or elasticity. In volume rendering, the voxels are displayed and manipulated directly with computers having substantial amounts of memory and processing power.
Advantages of Volume Rendering Include:
a) the ability to display the 3D volumes with no knowledge of the volume data set and hence no need to transform the data to an intermediate surface representation;
b) the ability to display any part, including internal structures, since the entire volume has been preserved; and
c) less susceptibility to discontinuities in the 3D scanning since the underlying volume is maintained.
Disadvantages of Volume Rendering Include:
a) the need for computers with a large memory and a great deal of processing power since the entire volume is displayed and manipulated; and
b) much slower rotations and manipulations are likely even with a large memory and substantial processing power.
One method which somewhat reduces the vast amount of data storage and processing connected with volume rendering and processing is known as octrees. An octree representation of a volumetric image is based on an hierarchial volume subdivision where each volumetric cube is broken into eight equal, sub-cubes. Each of these sub-cubes in turn can be broken into eight sub-cubes of its own. Described in parent-child nomenclature, if all children of an octree branch are included in the graphical image of the volume, then only the parent data need be recorded or manipulated, representing an 8:1 reduction in data and computation time. If two generations of octree levels are included by reference to a grandparent then a 64:1 reduction in data and computation time occurs. This approach maintains the fine resolution for an edge at the child level but enables efficient manipulation when grandparent or great-grandparent cubes of data are in common. This hierarchial level treatment may be extended to any number of generations. With specially derived computation methods volume unions, intersections, and manipulations are much more efficient than brute force treatment of all individual voxels. In the case of MRI or ultrasound data, each child, parent or grandparent cube element may be assigned characteristics such as density in addition to position.
1.6. Thermal Imaging
Thermal Imaging Technology
Historically, thermal imaging equipment was large, inconvenient and expensive. It yielded analog display information with the use of detection elements which required cooling via a supply of liquid nitrogen. Large battery packs were required for any attempt at portable operation. Costs for such a camera system were $50,000–60,000.
Recent solid state developments have resulted in thermal imaging cameras that are only slightly larger that a 35 mm photographic camera. They do not require cooling and easily operate at room temperature. One such thermal imaging camera is the IR SnapShot® manufactured by Infrared Solutions, Inc. This camera is based on Honeywell infrared (IR) thermoelectric thermal array detector technology. It is an imaging radiometer, an infrared camera that acquires a thermal image of a scene and can determine the temperature of any pixel within that scene. Pressing a push button on the camera causes a 120-element linear thermoelectric detector array to scan across the focal plane of a germanium IR lens in approximately 1.5 seconds. Software within the camera permits the 120.times. 120 pixel thermal images to be stored in flash memory cards or downloaded directly to a laptop or desktop computer for processing. The calibrated thermal images may be displayed with numerous colormaps on either the color LCD display of the camera or on the computer displays.
Radiometric IR cameras that operate at a video rate are nearing the end of development. Such cameras promise the thermal accuracy of the still IR cameras with image acquisition at the faster video rate.
Thermal images from radiometric cameras such as those described above provide a wealth of thermal information which can be analyzed and processed. The data is basically a matrix of temperatures in which each element corresponds to a pixel in the thermal image. It is common for IR camera manufacturers to provide software which computes thermal histograms of the scene and user selectable area or line indicators which then provide thermal properties of the selected area or line region of the image.
Thermal Imaging as an Indicator of Backfat
Driven by consumer desire for leaner meat products, there is application in the livestock industries for accurate and convenient methods to evaluate fat content or lean:fat ratios. While the total dissection of muscle mass is still the most accurate method, livestock producers and processors have long measured backfat thickness via ultrasound or directly as an indication of lean:fat ratios. A number of inventors have attempted to employ noninvasive thermal imaging to obtain an indication of lean:fat ratios and other meat quality measurements.
1.7. Calculation of Volumetric Measurements
Silhouette (Profile) vs. 3D Calculations of Volume
In order to evaluate the need for three-dimensional data techniques, it is of value to consider the calculation of volume for a standard geometric shape such as a cylinder.
First consider computing the volume of a cylinder from one or several side views. The diameter and length of the cylinder are D and L, respectively. From any side view, a silhouette or profile approach sees a rectangle that has width, D, and length, L. Any attempt at estimating volume from silhouette data would yield a cylinder volume of:Vsilhouette=D2L  (1-1)
where Vsilhouette is the volume of the cylinder using silhouette data; D is the cylinder diameter; and L is the cylinder width.
Considering the same cylinder from side views with 3D data yields the true cylinder volume:V3D=πD2L/4  (1-2)ERROR=Vsilhouette/V3D−1=D2L/πD2L/4−1=4/π−1=27.3%;  (1-3)
Considering the same cylinder from side views with 3D data yields the true cylinder volume:
                                          V                          3              ⁢              D                                =                                    π              ⁢                                                          ⁢                              D                2                            ⁢              L                        4                          ;                            (                  1          ⁢                      -                    ⁢          2                )            
with variables as defined previously.
To evaluate the error of the volume calculation using silhouette or profile in formation:
                              ERROR          =                                                                      v                  silhouette                                                  V                                      3                    ⁢                    D                                                              -              1                        =                                                                                                      D                      2                                        ⁢                    L                                                                              π                      ⁢                                                                                          ⁢                                              D                        2                                            ⁢                      L                                        4                                                  -                1                            =                                                                    4                    π                                    -                  1                                =                                  27.3                  ⁢                  %                                                                    ;                            (                  1          ⁢                      -                    ⁢          3                )            
A natural defense for the above error estimation is that a silhouette view from the end of the cylinder would acquire the necessary circular data. However, in the evaluation of livestock, most silhouette methods use only side and top. Logistically, an end view requires that a camera be placed directly in the path of the animal and an opposing wall be placed at the opposite end. A second practical consideration is that such an end view, in profile, would not be accurate if the animal axis was slightly skewed in one direction or the other. The profile would also be compromised if the head of the animal was turned to one side.
From the above considerations it is evident that a true three-dimensional imaging system will more accurately represent the volume of an animal than silhouette or profile systems.